On Lyapunov Exponents for RNNs: Understanding Information Propagation Using Dynamical Systems Tools

Recurrent neural networks (RNNs) have been successfully applied to a variety of problems involving sequential data, but their optimization is sensitive parameter initialization, architecture, and optimizer hyperparameters. Considering RNNs as dynamical systems, natural way capture stability, i.e., the growth decay over long iterates, are Lyapunov Exponents (LEs), which form spectrum. The LEs bearing on stability RNN training dynamics since forward propagation information related backward error gradients. measure asymptotic rates expansion contraction non-linear system trajectories, generalize analysis time-varying attractors structuring non-autonomous data-driven RNNs. As tool understand exploit dynamics, spectrum fills an existing gap between prescriptive mathematical approaches limited scope computationally-expensive empirical approaches. To leverage this tool, we implement efficient compute for during training, discuss aspects specific standard architectures driven by typical datasets, show that can serve robust readout across With exposition-oriented contribution, hope draw attention under-studied, theoretically grounded understanding in